Structure-from-motion for Calibration of a Vehicle Camera System with Non-overlapping Fields-of-view in an Urban Environment

Vehicle environment cameras observing traffic participants in the area around a car and interior cameras observing the car driver are important data sources for driver intention recognition algorithms. To combine information from both camera groups, a camera system calibration can be performed. Typically, there is no overlapping field-of-view between environment and interior cameras. Often no marked reference points are available in environments, which are a large enough to cover a car for the system calibration. In this contribution, a calibration method for a vehicle camera system with non-overlapping camera groups in an urban environment is described. A-priori images of an urban calibration environment taken with an external camera are processed with the structure-frommotion method to obtain an environment point cloud. Images of the vehicle interior, taken also with an external camera, are processed to obtain an interior point cloud. Both point clouds are tied to each other with images of both image sets showing the same real-world objects. The point clouds are transformed into a self-defined vehicle coordinate system describing the vehicle movement. On demand, videos can be recorded with the vehicle cameras in a calibration drive. Poses of vehicle environment cameras and interior cameras are estimated separately using ground control points from the respective point cloud. All poses of a vehicle camera estimated for different video frames are optimized in a bundle adjustment. In an experiment, a point cloud is created from images of an underground car park, as well as a point cloud of the interior of a Volkswagen test car is created. Videos of two environment and one interior cameras are recorded. Results show, that the vehicle camera poses are estimated successfully especially when the car is not moving. Position standard deviations in the centimeter range can be achieved for all vehicle cameras. Relative distances between the vehicle cameras deviate between one and ten centimeters from tachymeter reference measurements. 1. DRIVER OBSERVATION FOR DRIVER INTENTION RECOGNITION One of the big goals in the automotive industry is to reduce the number of traffic fatalities to zero (Volvo Vision 2020 (Samuelsson, 2017)). An important part on this way is to know the intention of a car driver for the next seconds in advance. Currently available cars are therefore equipped with environment cameras to collect information about other traffic participants around the own car. For example, environment cameras can capture a preceding car slowing down on the rightmost lane and having activated the right turn indicator. Advanced driver assistance systems can use driver intention recognition algorithms to anticipate, that the preceding car driver wants to turn right. To anticipate the intention of the own car driver, interior cameras observing his behavior (figure 1) can be used in addition. Their images can be used to extract features about the driver’s head movement and his gaze direction. The intention recognition can be made more reliable, if information from the environment and interior cameras is combined. For example, combined evaluation allows to evaluate, whether the driver has noticed the slower car in front of him or is distracted by the car radio, for example. Therefore, his head orientation and gaze direction have to be linked to the position of the other car relative to his car. Basis for this geometric link is the known relative position and orientation (pose) of the interior and exterior vehicle cameras to each other. A system calibration for the vehicle cameras can be used to estimate these parameters. For calibration of a vehicle camera system, a calibration environment large enough to contain a car is required, as well as the Figure 1: A driver camera can be used to extract features from images to recognize the driver intention of the next seconds as an important milestone to increase traffic safety (Volvo, 2014). car has to be able to drive into this environment. Urban structures, like parking garages or court yards, can be used for this purpose. On the one hand, in urban structures additionally placed photogrammetric reference marks cannot be used, as these structures are public or there is no permission from the owner. On the other hand, vehicle interior cameras can see almost nothing than the interior space of a car. Large parts of the interior of a car, like the window pane or the seats cannot be used to place reference marks. Therefore, a calibration approach using unmarked reference points is required. For example, feature points extracted from images and their 3d coordinates calculated automatically can be used as reference. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-1/W1, 2017 ISPRS Hannover Workshop: HRIGI 17 – CMRT 17 – ISA 17 – EuroCOW 17, 6–9 June 2017, Hannover, Germany This contribution has been peer-reviewed. doi:10.5194/isprs-archives-XLII-1-W1-181-2017 181 3d coordinates of reference points can be calculated automatically by triangulating image features of urban structures shown in multiple images. However, as the vehicle interior cameras are showing nearly only the interior space of the car, reference information of the urban structures is not available for them. Other reference information has to be used for these cameras, and the reference information for both environment and interior cameras has to be linked together for a system calibration. As costs are a very important factor in the automotive industry, the number of cameras is kept small. This leads in addition to non-overlapping fields-of-view, making the camera system calibration using tie points in overlapping image parts impossible. Due to their wide field-of-view and their low costs, so called “action cameras” can cover a huge part of the environment around a car despite their small number. Therefore, it has to be considered, that the wide field-of-view might cause large image distortions requiring reliable single camera calibration. In addition, the camera mounting on the window panes of the car might not be rigid over time caused by mechanical movements during a car drive. To check the vehicle camera poses again and again, the system calibration has to be repeated from time to time, requiring a calibration process which can be performed with low effort, for example before and after every video recording drive. The camera system calibration provides information about the relative poses of the vehicle cameras. To know, whether a pedestrian shown in one of the environment camera images is behind or in front of the own car, the camera poses have to be linked to the pose of the car. This can be done by transforming the camera poses into a vehicle coordinate system, which describes the movement direction of the car. As ground control points for the transformation, physical points on the car surface corresponding to the vehicle coordinate system have to be chosen.